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ABSTRACT 

Contingency tables, and their associated statistical 
tests» are frequently used in educational and social research. 
Popular statistical tests used in contingency table analyses include 
the Pearson chi- square test and the likelihood ratio chi-square test. 
These two t'^sts are chi-square distributed undor large sample 
conditions. However, when a given contingency table includes small 
cell frequencies the obtained chi-square statistic may not be 
asymptotically valid, and the associated probability values will be 
inappropriate. An alternative statistic which has been recommended 
for small samples or sparse conditions is the power-divergence 
statistic (Read ^ Cressie, 1988). This study was an investigation of 
the performance of three tests of independence for I x J contingency 
tables under small sample conditions. Specifically, the objectives of 
the research were to investigate the power and Type I error rates 
under small sample or sparse conditions of the Pearson chi-square 
test, the likelihood ratio chi-square test, and the Read and Cressie 
power-dive? gence statistic with lambda = 2/3 (Read Cressie, 1988). 
The power and Type I error rates were estimated for a variety of 
table dimensions, marginal distributions, sample sizes, and effect 
sizes. (Contains 16 references, 4 figures, and 4 tables.) (Author) 
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Comparison of Statistical Tests of Independence for 
Sparse I x J Contingency Tables 

Abstract 

Contingency tables, and their associated statistical tests, are frequently used in educational and 
social research. Popular statistical tests used in contingency table analyses include the Pearson chi- 
square test and the likelihood ratio chi-square test. These two tests are chi-square distributed, under 
large-sample conditions. However, when a given contingency table includes small cell frequencies 
the obtained chi-square statistic may not be asymptotically valid, and the associated probability 
values will be inappropriate. An alternative statistic which has been recommended for small 
samples or sparse conditions is the power-divergence statistic (Read & Cressie, 1988). 

This study was an investigation of the performance of three tests of independence for I x J 
contingency tables under small samole conditions. Specifically, the objectives of the research were 
to investigate the power and Type I error rates under small sample or sparse conditions of the 
Peai'son chi-square test, the likelihood ratio chi-square test, and the Read and Cressie power- 
divergence statistic with X = 2/3 (Read & Cressie, 1988). The power and Type I error rates were 
estimated for a variety of table dimensions, marginal distributions, sample sizes, and effect sizes. 
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Comparison of Statistical Tests of Independence for 



Sparse I x J Contingency Tables 



Introduction 



Conlingency tables, and their associated statistical tests, are frequently used in educational and 
social research. Popular statistical tests used in contingency table analyses include the Pearson chi- 
squarc test of independence and the likelihood ratio chi-square test of independence. These two 
tests lire asymptotically equivalent and, under large-sample conditions, are chi-square distributed. 
However, when a given contingency table includes small cell frequencies, the obtained chi-square 
statistic may not be chi-square distributed. Under these conditions, when the statistic is not 
asymptotically valid, the associated probability values will be inappropriate. If an obtained statistic 
is not distributed according to the chi-square distribution, researchers may mistakenly base 
decisions on inaccurate information. They may reject a null hypothesis when it is true, or fail to 
reject a null hypothesis when it is false. 

Several statistics have been proposed for testing independence between variables in a two 
dimensional contingency table. Three of these statistics were investigated in this research. 

The Pearson chi-square test is given below, where O refers to the observed frequency of each 
cell and /: to the expected frequency. The summation is over the cells in the contingency table. 
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The likelihood ratio chi-square test is asymptotically equivalent to Pearson's chi-square test 
(Fienberg, 1977), and is often used with larger contingency tables, because of its facility with model 
testing and chi-square partitioning (Upton, 1978). This statistic is computed as: 



Various corrective actions may be taken by researchers when contingency tables have small 
overall samples or some sparse cells. One corrective action is to collapse across categories of one 
or more of the research variables. However, this may lead to categories of lesser meaning than the 
original variable held. A second action may be to use exact methods to compute the probability for 
a given table, rather than to base probability on the tabled chi-square distribution (Agresti, 1990). 
For larger I x J tables, this can be computationally prohibitive. Another recommended course of 
action has been the use of alternative statistics (e.g., Kroll, 1989; Richardson, 1990). Read and 
Cressie (1988) have suggested the following statistic as one which is less susceptible to the effects 
of sparseness than either or G^. 
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In their unified model, Cressie and Read (1984), and Read and Cressie (1988), suggest that in 
tact Pearson's chi-square (with X set equal to 1) and the likelihood ratio chi-square (the limit as X 
approaches 0) are just two members of a family of power-divergence statistics, all of which can be 
represented by the formula above. Read and Cressie (1988) have found the power-divergence 
statistic with X = 2/3 lo have some excellent properties, including optimal performance under small 
sample conditions. 

Purpose 

The Type I error estimates for small sample chi-square statistics (both and G^) have been 
investigated for 2 x 2 tables (Camilli & Hopkins, 1978, 1979; Larntz, 1978; Roscoe & Byars, 1971; 
Thompson, 1988), Considerably less research has been conducted on relative power, or on tables 
of larger dimensions (Mehta & Hilton, 1993; Parshall & Kromrey, 1994). However, it is often the 
case that as table size increases, so does the probability of at least some very low frequency cells 
(Read & Cressie, 1988). 

The purpose of this study was to investigate the performance of three tests of independence for I 
X J contingency tables under small samples and sparse conditions. Specifically, the objectives of 
the research to be reported were to investigate the power and Type I error rates under small sample 
and sparse conditions of the Pearson chi-square test, the likelihood ratio chi-square test, and the 
Read and Cressie statistic with X = 2/3. The power and Type 1 error rates were estimated for 
various table dimensions, marginal distributions, s'Miiple si/cs, and effect si/.cs. 
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Educational Importance of the Study 



Because of the frequent application in educational research of contingency table analyses, the 
operating characteristics of statistical tests of independence is an important area of inquiry. 
Similarly, the use of small samples is often necessary in educational research. Knowledge of the 
power and expected Type I error rates of tests of contingency tables in which small samples arc 



The research to be reported was a Monte Carlo study. Random samples were generated for a 
series of I x J contingency tables: 2 x 5, 3 x 5, 5 x 5, and 2 x 7, 3 x 7, 5 x 7. For each table 
dimension, three conditions of population marginal distributions were examined: equal marginals, 
slightly skewed, and highly skewed. Small, medium, and large population effect sizes were 
examined. In Cohen's (1988) power analysis text, these effect sizes correspond to w values of 0.10, 
0.30, and 0.50, respectively. Cohen's effect size w is computed as: 



involved (either for the total table or for some subset of cells) will help inform researchers who arc 



planning studies in which contingency table analyses will be used. 



Method 
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Where Po, is the expected proportion in cell /, and Pn is the observed proportion in cell /. hi 
addition to these ct'fect sizes, a null model (no effect, or no dependency between the two marginal 
variables) was also examined, to e\'iiluate the extent to which Type I error rates match nominal 
alpha levels. 

Programming for (he Monte Carlo Study 

The programming for the Monte Carlo study was written in SAS version 6.06. The data were 
generated using uniform random numbers on the zero to one interval (the SAS RANUNI function). 
To simulate samples for a test of independence, a separate series of random numbers was generated 
for each row of the contingency table, with each row consisting of the same number of 
observations. The obsei*vations were then assigned to columns in the table based upon the value of 
the random number. 

. For example, with a 2 X 5 table with equal marginals and an effect size of zero, two series of 
random numbers were generated. Observations with random numbers between zero and .20 were 
assigned to the first column of the contingency table, those with random numbers between .20 and 
.40 were assigned to the second column, etc. This procedure yields tables in which the expected 
proportion in each cell is .10, each column marginal proportion is expected to be .10, and each row 
marginal proportion is fixed at .50. 

The column marginal proportions of the tables examined in the study were controlled by 
assigning larger or smaller ranges of the uniforni random numbers to each column. Por example, 
for a 2 X 5 table with 60:10:10:10:10 column marginals and an effect si/e of zero, observations 
with random luimlx^rs between zero and .60 were assigned to the first column of the contingency 
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table, those with random numbers between .60 and .70 were assigned to the second column, those 
with random numbers between .70 and .80 were assigned to the third column, those with random 
numbers between .80 and .90 were assigned to the fourth column, and those with random numbers 
higher than .90 were assigned to the fifth column. As with the equal marginals procedure described 
above, the tables produced with this procedure have cells with expected proportions that are equal 
to the products of the table's marginal proportions. 

Three column marginal distributions were examined in this study. The equal marginal condition 
provided equal proportions at each level of the column variable. A slightly skewed maiginal 
distribution was produced by generating tables in which the expected value of the first column of 
the contingency table was 60% of the data, and the remaining 40% was evenly dispersed over the 
other columns of the table (i.e., for a five-level column variable each level was expected to receive 
40/4 or 10% of the obsei-vations; for a seven-level column variable each level was expected to 
receive 40/6 or approximately 6.67% of the data). Similarly, a more highly skewed column 
marginal was produced by generating tables in which the expected value of the first column was 
80% of the data and the remaining 20% was evenly distributed over the remaining columns. 

Finally, non-null effects were generated by assigning observations to table cells in proportions 
that differed from the products of the table's marginal proportions. For example, for a 2 X 5 table 
with equal m^u'ginals and an effect size of .50, observations from the first row with random 
numbers between zero and .312 were assigned to the first column of the contingency table, those 
with random numbers between .312 and .624 were assigned to the second column, those with 
random numbers between .624 and .712 were assigned to the third cohimn, tliose with random 
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numbers between .712 and .800 were assigned to the fourth column, and those with random 
numbers larger than .800 were assigned to the fifth column. The procedure was reversed for the 
random numbers representing the second row of the tabic. For these data, random numbers between 
0 and .088 were assigned to the first column, those with random numbers between .088 and .176 
were assigned to the second column, those with random numbers between .176 and .488 were 
assigned to the third column, those with random numbers between .488 and .800 were assigned to 
the fourth column, and those with random numbers higher than .800 were assi2ned to the fifth 
column. This procedure yields tables in which the expected proportion in each cell of the first four 
columns deviates by .056 from the product of the marginal proportions, corresponding to Cohen s 
(1988) w of .50. 

For each size of contingency table investigated, a total of five sample sizes were produced. For 
the 2 X J tables, overall sample sizes of 16, 32, 64, 128, and 256 were studied. For the 3 X J tables, 
sizes were 18, 33, 66, 129, and 258. For the 5 X J tables, the sample sizes investigated were 15, 30, 
65, 130, and 255. For each table size and sample size, the total sample size was equally divided 
over the rows in the contingency table. Five thousand samples of each size were drawn under each 
of the experimental conditions examined. The use of five thousand replications provide maximum 
95% confidence intervals of ± .014 around the observed proportion of null hypotheses rejected 
(Robey & Barcikowski, 1992). 

1^'or each condition, three test statistics were computed: (a) Pearson's chi -square test of 
independence, (b) the likelihood ratio chi-square test, (c) and the Read and Crcssic power- 
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divergence statistic with X = 2/3. Estimates of the statistical power of each test were conducted at 
alpha levels of .01, .05, and .10. (However, to limit the number of figures included, only the results 
for the nominal alpha level of .05 are presented in figures.) 

Results 

The empirical estimates of the Type I error rates for the three statistical testing procedures arc 
presented in Table 1 for the conditions examined in this study. At a .01 nominal alpha level, 
both the Pearson chi-square and the Read and Cressie statistic were consistently conservative in 
Type I error control for the smallest samples examined. For the samples of size 16, the estimated 
Type I error rates ranged from zero to only .003 for the Pearson chi-square and to .004 for the 
Read and Cressie statistic. As the sample sizes increased, the estimates of Type I error rates of 
both procedures approached the nominal alpha level. However, for the conditions involving 
asymmetric marginal distributions, these tests remained somewhat conservative even with the 
largest samples examined in this study. For example, with the 3 X 7 tables and samples of size 
258, the estimates of Type I error rates were .005 for both of these tests in the extreme 
asymm.etry condition. In contrast to the conservatism shown by the Pearson chi-square and the 
Read and Cressie statistics, the likelihood ratio chi-square showed a marked tendency to be 
excessively liberal in Type 1 error control. Using Bradley's (1975) liberal criterion of statistical 
robustness (i.e., estimated Type 1 error rate within the range of Otionunai ± -5 cx^uMnmai). the likelihood 
ratio chi-square exceeded these limits in 38 of the 90 conditions examined with a nominal alpha 
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level ot .01. The most extreme Type I error rate was seen in the 5X7 tables with equal 
marginals and san^iples of size 65, in which the estimate of the Type 1 error rate for the test was 
.04 K n-iore than four times the nominal alpha level. 

Insert Table I About Here 

The results were consistent with non^iinal alpha levels of .05 and .10, although as should be 
expected the Type I error rates were closer to nominal levels for these more liberal alpha levels. 
The Pearson chi-square and the Read and Cressie statistic remained conservative with small 
samples and asymmetric marginals at both of these nominal alpha levels. The likelihood ratio 
chi-square remained liberal in many of the conditions examined, exceeding Bradley's liberal 
limits of robi Uness in 44 of the 90 conditions examined at a nominal alpha level of .05 and in 
47 of the 90 conditions at a nominal alpha level of .10. 

Of interest to note in Table 1, the relative conservatism of the Pearson chi-square and the 
Read and Cressie statistic depended upon the table size, with the Read and Cressie statistic being 
less conservative than the Pearson chi-square in the smaller tables, but more conservative in the 
larger tables. For example, at the .05 nominal alpha level in the 2 X 7 tables with equal 
marginals and samples of size 16, the estimated Type I error rate for the Read and Cressie 
statistic was .027 while that of the Pearson chi-square was .017. With the smallest sample size in 
the 3 X 5 and 3X7 tables with equal marginals, however, these tests were nearly equally 
conservative (.032 vs. .033 for the 3 X 5 tables, and .021 vs. .022 for the 3 X 7 tables). In 
contrast, for the 5 X 5 and 5X7 tables with the smallest sam[)les and equal marguials, the 
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Pearson clii-squarc was less conservative than the Read and Cressie statistic (.027 vs. .014 for the 
5 X 5 tables, and .021 vs. .006 for the 5 X 7 tables). Regardless of table dimensions^ however, 
with larger sample sizes the estimated Type I error rates of these two statistics converged (see 
Figures 1 and 2). 

Insert Figures 1 and 2 About Here 



Estimates of statistical power are provided in Tables 2 and 3. Because the likelihood ratio 
chi-square did not adequately control Type I error rates for the sparse tables examined in this 
study, power estimates for this test are not included. Table 2 provides power estimates averaged 
across effect sizes, while detailed results are provided in Table 3. 

Insert Tables 2 and 3 About Here 



An examination of Table 2 shows that the average power differences between the Read and 
Cressie statistic and the Pearson chi-square were small. However, the Read and Cressie statistic 
was consistently more powerful, on average, than the Pearson chi-square for the smaller tables 
examined, while the Pearson chi-square evidenced power advantages for the larger tables. Such 
power differences, however, were found only with the smaller sample sizes. With the larger 
sample sizes examined, the two tests showed similar power across conditions examined. This 
pattern of power differences is consistent with the pa; tern of Type I error rate estimates presented 
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in Table L with each lest having lower power for those table sizes in which the test is the more 
conservative in terms of Type I error control. 

For example, at the nominal alpha level of .01, the greatest difference favoring the Read and 
Cressie statistic was obtained in the 2X7 tables with samples of size 64 and extreme skewness 
in the marginals. In this condition, the power of the Read and Cressie statistic was .230, while 
thai of the Pearson chi-square was . 179, a difference of. 05 1 . The greatest difference favoring the 
Pearson chi-square was obtained in the 5 X 7 tables with samples of size 130 and extreme 
marginal skewness. In this condition, the estimated power of the Pearson chi-square was .285, 
while that of the Read and Cressie statistic was .242, a difference of .043. The Read and Cressie 
statistic provided slight but consistent power advantages relative to the Pearson chi-square for the 
2 X 5 and 2 X 7 tables. 

Similarly, for the nominal alpha level of .05, the greatest difference in power estimates 
favoring the Read and Cressie statistic was seen in the 2X7 tables with samples of size 32 and 
extreme marginal skewness, where the power of the Read and Cressie statistic was .124 while 
that of the Pearson chi-square was .087, a difference of .037. The greatest difference favoring the 
Pearson chi-square was obtained in the 5 X 7 tables with samples of size 65 and extremely 
skewed marginals. In this condition, the estimated power of the Pearson chi-square was .163, 
while that of the Read and Cressie statistic was .099, a power difference of 064. 

Finally, for the nominal alpha level of .10, the greatest power difference favoring the Read 
and Cressie statistic was seen in the 2 X 7 tables with samples of size 32 and extreme marginal 
skewness. In this condition the estimated power of the Read and Cressie statistic was .250 while 
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that of the Pearson chi-square was .203, and a difference of .047. Conversely, the greatest 
estimated power difference favoring the Pearson chi-square was obtained in the 5X7 tables with 
samples- of size 65 and extreme skewness in the marginal distributions. In this condition the 
power of the Pearson chi-square was .256 while that of the Read and Cressie statistic was .175, a 
difference of .081. 

In an examination of the more detailed results on the statistical power estimates presented in 
Table 3, it should be noted that for those conditions that showed the greatest overall power 
differences, the differences increase with the effect size of the condition. For example, at a 
nominal alpha level of .05, with the 2 X 7 tables with extreme skewness and sample size 32, both 
tests showed almost no power at the small effect size (.004 for the Pearson chi-square and .006 
for the Read and Cressie statistic). For the medium effect size (0.3) the estimated power of the 
Pearson chi-square was .043 while that of the Read and Cressie was .061. Finally, for the large 
effect size (0.5) the power estimates were .213 and .304, for the two tests respectively (see Figure 

3) . Similarly, for the 5 X 7 tables with extreme skewness and samples of size 65, under a 
nominal alpha of .05, the Pearson chi-square and the Read and Cressie statistic showed power 
estimates for the small effect size of .022 and .007, respectively. With a medium effect size the 
power estimates were .099 for the Pearson chi-square and .049 for the Read and Cressie statistic, 
wile for the large effect size the power estimates were ,369 and .240 for the two tests (see Figure 

4) . 

Insert h'igiircs 3 and 4 About Here 
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A summary of the power comparisons is presented in Table 4. This tabic presents the number 
and percent of conditions in which the Pearson chi-square was nominally more powerful than the 
Read and Crcssie statistic, the number and percent of conditions in which the Read and Crcssie 
statistic was more powerful, and the number and percent of conditions in which the power 
estimates were identical. For example, at a nominal alpha level of ,01, for the 2X3 and 2X7 
tables, the Read and Cressie statistic was more poweffi'.l than the Pearson chi-square in 73% of 
the conditions examined, and this ttst was not less powerful than the chi-square in any of the 
conditions examined. For the 3 X 5 and 3X7 tables, the tests showed identical power in 40% of 
the conditions, the chi-square test was more powerful in 33% of the conditions and the Read and 
Cressie statistic was more powerful in 27% of the conditions. For the 5 X 5 and 5X7 tables, the 
chi-square test was more powerful than the Read and Cressie statistic in 78% of the conditions. 
As may be seen in Table 4, this pattern was evident across nominal alpha levels. However, for 
the 3 X 5 and 3X7 tables, the power estimates for the Read and Cressie statistic exceeded those 
of the Pearson chi-square at nominal alpha levels of .05 and AO more frequently than it did for 
the nominal alpha level of »0L 

Insert Table 4 About Here 



Discussion 

The likelihood ratio test demonstrated a poor ability to maintain the nominal Type I error rale, 
disciualifyinp it for further consideration. (A related, earlier investig:ilion of 2x2 and 2x3 
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contingency table analyses found the likelihood ratio test to display relatively poor power along 
witli liberal Type I error rates compared to other statistics investigated. See Parshall & Kronirey, 
1994.) The performance of the remaining, competing statistics in this study converged as sample 
size increased, regardless of the size of the table. On a practical basis, this means that for the 
conditions studied, once a sample is large enough it may be a matter of indifference to the 
practitioner whether Pearson's chi-squiu'e or the Read and Cressie statistic is used. 

With the smaller samples however, differences in the performance of the two statistics were 
found. The Read and Cressie statistic displayed greater power than Pearson's statistic in the smaller 
tables, while Pearson's chi-square test out-performed the Read and Cressie statistic in the larger 
tables. While both tests demonstrated increased power as the effect size increased, this was not a 
general improvement. Rather, the Read and Cressie statistic displayed a greater increase in power 
associated with increased effect size in just those smaller tables where it already displayed a 
performance advantage over Pearson's test, while Pearson's test demonstrated a greater power 
increase associated with increased effect size in the larger tables where it displayed better 
performance. Both statistics are more powerful under equal marginals conditions. Both also 
demonstrate a marked conservativeness in the tables with highly skewed marginals, although the 
general pattern of better performance of the Read and Cressie for smaller tables and better 
performance for the Pearson for larger test hold here as well. 

Read and Cressie (1988) point out a number of the variables and assumptions which may need 
to be considered, including ''the sample size the number of cells the form of the null model 
(loglincai, etc.), and the ^direction of departure' ot the alternative from the null" (pg. 80). Their 
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model for analysis suggests using the family of power-divergence statistics, selecting the particuhir 
value of lambda which is optimal under given conditions. For example, they indicate that 
Pearson's chi-square (lanibda=l) may be optimal when expected ceil frequencies are near equal 
across a table, and when the number of table cells is at least 20, and G'^ is optimal for certain 
nonlocal alternatives with a limited number of near-zero probabilities. When the expected cell 
frequencies arfe unequal, they recommend X = 2/3, specifying that Pearson's chi-square statistic can 
display serious bias for sparse tables with unequal cell probabilities. 

For the conditions examined in this study, it would appear that a decision regarding the best 
statistic to apply in a given situation needs to be based on table size as well as sample size. 
(Skewness also impacted the results, by lessening power and increasing conser\'ativeness, but this 
effect was consistent across statistical test.) In general, as Read and Cressie suggest, X = 2/3 may 
be a good compromise solution when a researcher has little knowledge about possible alternative 
hypotheses. Conversely, a researcher may opt to use Pearson's test under large table, small sample 
conditions, the Read and Cressie power-divergence statistic under small table, small sample 
conditions, and either statistic with large samples. 

Future research should include a more detailed investigation into sparseness. One aspect of 
this study which may limit the generalization of results concerns the manner in which data were 
generated. The data were simulated according to expected marginal proportions (e.g., equal 
marginals, slightly skewed, and highly skewed marginals), while the total table sample size was 
held near constant. For example, a 5 x 7 table with sample size of 236 will be sparser than a 2 x 7 
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table with the same total sample size. Thus, as table size increased, simultaneously sparsencss also 
increased. A follow-up study could alter this data-generaiion design to provide more information 
about the interaction of sample size with table size.. Additionally, other approaches for modeling 
the alternative hypothesis could be simulated. While one reasonable alternative was modeled in the 
current study, a given contingency table can differ from the null in many ways, potentially affecting 
the selection of the optimal statistical test. 
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Table 4 



Number and Percent of Conditions with Differences in Empirical Power Estimates by Table Size. 



Nominal Alpha = .01 



Nominal Alpha = .05 



Nominal Alpha = .10 



Table 
Size 



Power Differences 



RC>Chi2 No Diff Chi2>RC 



Power Differences 



RC>Chi2 No Diff Chi2>RC 



Power Differences 



RC>Chi2 No Diff Chi2>RC 



2X5 



33 
73% 



12 
27% 



0 

0% 



37 
82% 



8 

18% 



0 

0% 



33 
73% 



12 

ITU 



0 

0% 



2X7 



33 
73% 



12 
27% 



0 

0% 



38 
8A% 



7 

16% 



0 

0% 



36 
80% 



9 

20% 



0 

0% 



3X5 



12 
27% 



18 
A0% 



15 
33% 



26 
58% 



13 
29% 



6 

13% 



30 
67% 



6 

13% 



9 
20% 



3X7 



12 
27% 



18 
A0% 



15 
33% 



25 
56% 



8 

18% 



12 
27% 



20 
AA% 



7 

16% 



18 

A0% 



5X5 



2 

A% 



8 

18% 



35 
78% 



8 

18% 



9% 



33 
73% 



12 

27% 



6 

13% 



27 
60% 



5X7 



1 

2% 



9 

20% 



35 
78% 



5 

11% 



4 

9% 



36 
80% 



10 
22% 



3 

7% 



32 
71% 
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